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12.1) Ini ro Jutf iflit 

The percepl ron and the convergence theorems of Chapter II arc 
related to mart}' other procedures ihat are sludied in an extensive 
and disorderly literature tinder such titles as learning MALlttNES, 
MODELS DF LEARNING, INFOBMATIQN RETRIEVAL, STATISTICAL DE¬ 
CISION THEORV, PATTFSN Recognition and many more, In this 
chapter we will study a few of these to indicate points of contact 
with the perceptroni and to- reveal deep differences, We can give 
neither a fully rigorous account not a unifyiug theory of these 
topics: this would go as far beyond out knowledge y beyond the 
scope of this book. The chapter is written more in the spirit of 
inciting students to research than of offering solutions to pfotv 
lcins T 

12.3 Information Retrieval and Inductor Inference 
The peroeptron training procedures (Chapter 1 I) could bp Utedl lo 
construct a device that operates within the following pattern of 
behavior: 



Daring a ’"filing™ phase, the machine is shown a “data Ml” of 
w-cEimensiomri! vectors--one can ihirtk of them asufl-bil binary 
numbers or as points in n-space. Later, in a “finding" phase, the 
machine anist be able to decide which of a variety of “query" 
vectors belong to the data set. To generalise Lhis pattern w« will 
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uselhe term A (k far in algorithm that examines elements of a data _ 
net to modify the in far mutton in -a memory. A 1k is designed to pre¬ 
pare tiie memory far use by a nattier procedure, A r ** r which will 
use dhc informullort in the memory to make decisions about query 
vecl ors. 

This chapter will survey a variety of instances of this general 
scheme. We will begin try relating the pdrceptron procedure to 
the simplest such scheme: in the l-OMPLEIB storage procedure 
A hlf merely copies the drtts vectors, as they come, into the 
memory. For each query vector, A hod searches exhaustively 
through memory to see if it is recorded there. 

12, II Compn rinji fFRCKrtKON with LthirLCTt STOMCE 
Chir purpose is m illustrate, in tills simple ease, some of the ques¬ 
tions one might ask to compare retrieval schemes: 

Is tf\r procedure tiuhrr.fiiP The rerceftROM scheme works per¬ 
fectly only under lhe restriction that the data set is linearly 
separable. Covirlftf storage is universal: it works for any 
data set, 

flow niHch memorv is rPifitired? Complete storage needs a mem¬ 
ory large enough to hold the full data *et, Percept Ron. when it 
is ,applies Me, somdimes hus □ summarizing. effecl, in that the 
informiUKHi capacity needed to store hs Mcflickrtti (ffj is sub¬ 
stantially h:*.* than that needed to store the whole data SCI, We 
have seen (£1(1.3) IhjL this is not generally true; the coefficients 
for ^ PH1( , may heed much more storage than does the list of 
accepted vectors. 

IIow quickly dots Af„j operate? The retrieval scheme—exhaustive 
search ■ specified Tot coupr.t-FE STORAGE is very slow (usually 
slower than PKRCEPTROn 1 * A ll(d , which must also retrieve all its 
coefficients front memory}. On the olher hand, very similar pro¬ 
cedures could he much faster. For example, if Aj k did not just 
store Ihe data set in. its order of entry, but sorted the memory 
into numerical order, then A,„ d could use n binary search,. reduc-- 
ing the query-answer time to 


log 4 (Idata sell) 
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memory references, We shaTI study fin §11.6) A,-,* algorithms chal 
sacrifice memory she lo obtain drama Ire further increases in 
speed (by [lie so-called hosh-cadsug technique). 

Can the procedure operate urffA same degree (perhaps measured 
pnthabiiisifcaltyfof sttecess even when A,* fids seen attly a subset 
fifth? data set euIf il fl "dam MMph"? PERCFtTRON Ittight, but 
(he com pi ETF storage algorithm, as described, Clunoi make a 
reasonable guess ■when presented with a query vector not Ln the 
data Silihplc- This deficiency suggests an important modiJicution 
of the comptele storage procedure; tel Ai rJ , instead of merely 
checking whether (he query vector is in ftie data sample, find that 
member of the data sample closest to it. This would lead, on an 
a priori assumption about the "continuity" of the. data set, to a 
degree of generalization as good as the perceptron’s. Unfortu* 
natcly the speed-up procedures such as hash-ending cease to be 
available and we conjecture (in a sense to be (hade more precise 
in & I 2.7.6} that the loss ls irremediable. 

Olher considerations we shall mention concern the operation of 
A„ t , W< note that the PERCE PTItDN and (he COMPLETE STORaGeI 
pf(Kedu(cji share the follow-tag fcatiira: 

They act incre men t idly. that is, change the stored ircinory slightly 
us a function of the currently presented member of rhe data set. 

They Operate in ""real time " without using large quantities of 
auxiliary scratch-pad memory. 

The y can ncecpL_ Llic d am set in any order and arc tolerant of 
repetitions lhat cause only delay but Jo not change the final 
stule. 

On the other hand they dlifer in at least one very fundamental 
way' 

The perceptton's A..^ is a “search procedure" b ased on feedback 
from Us own r esults. The cOnipleU; storage file algorithm is pas¬ 
sive. The advantage for ihc pcrccplron is that under some condi¬ 
tions it finds an economical sii nun arising representutton. The cost 
is th:it it may need to see cadi data point tunny times. 

IM.l Multiple C’lassillcarlrMi Proeedunr^ 

H is -i slijihl gcncr.di/ation of these ideas to suppose the dala 
set divided into a number of classes F,..,, ,F*. The algorithm 
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A„u iis presented its before with members of the data set bill also 
with indications of the corresponding. daii. It constructs a body 
of sLored information which is handed over lo A lilt whose task 
is to assign query points to their classes using this information. 

Example: W C hjvc seen (|LI.3. t) how Lo extend Lhc concept of 
the perception to a multiple classification. The training algorithm, 
A^, finds A vectors Ai„...,Ai. and A rj ( assigns the vector ■* 
to F f if 

•9 ' Aj > $ + A, fall r j i* f). INNER PRODUCT 

Example: The fill lowing sil nation will seem much more familiar 
to rtany waders. If we thin* of each class F, as a "Clump'' 
Or "cloud” or "clutter” of points in $-space, Ihen we can imagine 
thiit with crich F, i* ajsodsted a special point B, that is; somehow, 
a “typical" or “avefaje" pomt. For example, R, could h e the 
cruirr of gravity, that is, iht mtm of all the voctora In Pj (or, 
sa>Lof just those vectors that SO fur have been observed to be in 
Fjh Then a familiar procedure is: # is Judged to be in that 
F, for which lhe Euclidean distance 

I* - ® A. 

is the itnaUcsi. Thru is, each <F is identified with the nearest B- 
point. 

Now this nearness scheme and the inner-product Scheme 
might look quite dilTitfenL but they are essentially the same! 
For we have only Lo observe that Lhe set of points closer to 
cine given point B ; chan to another B ; is bounded by ta hyper- 
plane (Figure L'.1), and hence can be defined by a linear inequal- 
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iL>-, Similnrly. Ihc polim closest U> one or a. number of B/s form a 
(convex’) puIjr&tiA (.Figure 12.2) and (his is true in higher dimen¬ 
sions, also, 



Fur [Hilly, we see this by observing that 

1 + - B.J 1 ™ l + l 1 - 2** R, + iBf. 

New, ir »e e-vn nssume Ihal all the 't'a bnvc ihe same length L then 
the biicliddP di<1aflCC(J?) WiM be xuralirit when 

*■ B. - J|B.,f “ * ^ - flj 

. ... I **" " ~ 

is largest. Bin this has «Mdl> the imier-producl. if the “threshold" 
if removed by §1.2.1 (I). Tn fee Lhul Ihe inner prudjcl concept .loies 
nolliing by requiring the t£i have llie fame lenglh, we add an eslra 
dimension and replace each 4 - 1^,. by 



mV 111 nl all i$‘s have tree III f. s - n. We have to add One dimunsspii In 
Lite l('s, [ao, but can always Set iis cocffiriunl 1* rcm. 

L2J A Variety of Classification Algorithms 

We select, from Ihe infinite variety of schemes that one might 
ItRC In divide a space into different climes, a few schemes that 
3 llLI.il rate ^spools of our mo in subject: eomputotvon and lioeaf 
separation. Wc vdll summarize each very briefly here; lh< ft- 
inainder of the chapter compares and contrasts some aspects of 
Iheir atgorilhmic structures, memory requirements, and cornmil- 
ments they make about the nature of the classes. 
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Each of our model* uses ike; same basic form of decision algo- 
rithtn for A rFj , In each ease there is assigned Lo each class F, 
one or more vectors A,; wc will represent I his assignment by say¬ 
ing that A, is associated wish Puri- Given a vector ■$-, the decision 
rule is always to choose that F,,., Tor which A, ■ * is largest. As 
noted in £12.1.2 this is Mathematically equivalent to a rule that 
minimises \'t - A, [ or some very similar formula. 

For each model we nsnst also describe the algorithm A Uf that 
eon*4rucls ihe A.'s. on ihe basis of prior experience, or a priori 
information about ihe classes In ihe brief vignettes below, the 
fi tit deta ilh of 4 lie A „ c procedures a re deferred to olhe r sect ions. 

12-2,1 TIml natcrmiON Procedure 

There is ore vector A,. Tor each class F^. A Fk ean he the procedures 
described in £ 11,1 for the 2-class case and in § l M t for (he multi' 
class ease. 

12,2.2 The bivfr. I drear ■Slnrislieaf Procedure 

Again we have one A, for ench F,. Am is quile different, however. 

For each class F, and cadi partial predicate <p„ define 



whorv p t{ i* ihr probability that * I. ghvn shut # is rb F,. Then 

fie hue 



We will explain in §12,4 3 the conditions under which Ibis use 
of “probability" makes SenM. and describe some "learning'* algo¬ 
rithm* 1 ha l could be itSed to estimate or approximate the iv^'s, 

The is.svrs procedure hus I he advantage that, provided certain 
statistical conditions are ^iPsfied. it gives good results foe cFasses 
that arcitot linearly Separable. In fact it gives the [owes! possible 
error rate for procedures in which A,j k depends only Cm condi¬ 
tional probabilities, given that Ltie are slntislicaliy independent 
in (be sense explained in I12.4.2. It is astounding that (his is 
achieved by n linear formula. 
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I1JJ1V BEST I'lAhKH Procedure 

In dirc/cjii situations either PEACtlPTA-ON or bates may he 
superior, Hul often, when thcF/Sane noL linearly separable. there 
will cjtisl a «i »r Aj vectors which will give fewer errors than 
cither of Lhese schemes So define the BEST fLanes procedure to 
use 1 hal wt of A/s for which choice of the largest A / - # gives (be 
Fewest error*, 

By definition, best FLAKES is always- an Feast as good as rayf* 
or peksfptaok, This does not corn rad ict the optimality of Bayes 
since the search for (he best plane uses information oLhcr than 
(be cohdilkmal probabilities. Unfortunately no practically effi¬ 
cient A lit is known for discovering its A/s. As noted in §11.3, 
hill-climbing will apparently not work welt because of Use local 
peak problem. 

11 . 1.4 flit [RODA TA PrOLrJurr 

lit ihc schemes described up to now, assigned csncLly one A- 
vecior £o each F-clatt. Sf «■( shift Lowaid the minimum-distance 
viewpoint, this suggest* that such procedures will wort satisfac¬ 
torily Only when the F-clusSoS arc “locaNrcd" into relatively iso¬ 
lated, single regions---one might think of dumps, clusters,, or 
clouds. Given this intuitive picture, one naturally asks what to do 
if an F-class, while not a neat spherical cluster, is nevertheless 
semifocaltiicd as a small number of clusters or, perhaps, a snaVtH 
like structure. We couEd still handle such situations,. using (he 
least-distance A (ril hv assigning an A-vector to «dl Subcluster 
of each F, and using several A‘s to outline the Spine of Lhe- snake, 
To real in* this concept, we ateed an- A^ scheme that has some 
ability ro analyse distribution* into clusters, W<; will describe one 
such scheme, called irijoata, in §12,5, 

1J-Z-S flic KFaPFsT MfiCdWiK I'roccdirrE 

Our simpFesI and most radicul scheme assumes no limit on the 
number or A-vectors. stores in mein pry every 4 that has ever 
been encountered, together with (he name of its associated F- 
elass. Given a query vector^, wc find which $ in the memory is 
closest to +n, and choose the F-elass associated with that 

fliis is o v^ry generally powerful arid hod: it is very efficient OA 
many sorts of cluster configurations.; it never makes a mistake on 
an already wen point; in the limit it approaches zero error 
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except isiiilcf i .iihe/ peculiar eircumseances (one of which is dis- 
eussedirt Ihc following section). 

xf*h.]=st icFtnHTHTE has an obvious disadvantage.—the very Large 
memory required a ad a subtle disadvantage: (here is reason to 
* aspect that it emails large, and fundamentally unavoidable, corni- 
put-alion costs (discussed in ■SI3--G), 

11,3 lllcnristic CconMtry of I.incur Seim redan Mettinds 
The diagrams of Shis section iiHenTpl lb suggest some of the be- 
havioral aspects of the nclhods described in §12.4 . To GOirtpeit" 
sale for our inability So depict multidimensional configurations, 
we ase iw.-u-d intension uL multivalued coordinates. The diagram* 
may appear plausible, bul they are really defective images that 
lIii iuh hint at the horrible things that con happen in a spare of 
many dimensions. 

Using this metaphorical tind of picture, we can suggest two kinds 
uf .situations which Lend to favor one or the other of SAVES or 
pchcefthosi Figure L2.3}. 



rlgutv (2.3 

The TIAYPS line in Figure 12.3 tends to lie perpendicular to she 
line between the “mean" points of f. and P t . Hence m Figure 
12.3(a), we find Lhai hay irS mates some errons. The sets art, irt 
fact L Siiv.:;nK separable, hence PEACE PTKON, evenLually, makes no 
error! Its all, In Figure 12.3(b) we find thut SAVES- maVei i few 
errors, justl as tn 12.3(a). Wc don't know much about rERdfiPTROb" 
in nonscparable situations; it is clear that in some situations it 
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will not Jo as well us bavfs. Dy definition BEST PLANE, of course, 
does at k;tS| as Well as either SAVES or PERCE PTRONi 

From the scan, the Wry suggestion that any of these procedures 
will be any good at all amounts to an a priori proposal that the 
F-tliite ctln be fitted into simple clouds of some kind, perhaps 
with a little overlappsng, as in Figure 12.4. Such an assumption 



Figure 12.4 

could be justified by some reason for believing Ihat the diiTcrence 
between f* and t are due to some one major Influence plus 
a variety of smaller, secondary effects. In general pehcepthon 
( endsto be sensitive to the outer boundaries of the clouds, and 
relatively insensitive Lo the density distributions inside, while 
BATES weights all +“s equally. Tn cases that do not satisfy either 
the single-cloud or the slight-overlap condition (Figure 12,5), we 
can expect BaVES to do badly, and presumably peeceptkon also. 
BEST PLaNL can be substantially better because it is- no| subject 
to |hy bad influence of symmetry Rut finding the best pi anr is 



v ( 


Figure 12 5 
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!Lk-±t> in involve bad eoliipi 11:1 Lion problems, because of itiu 
Locally optimal “bills.' 1 ' Figure 12.6 jhpws some of [he local [leaks 
for ns st pe.ans id the ease of a bad “parityltke” situation. Here, 
evert 15-OP at a *iv ill do badly Unlesi il is aflcw«d to have one A- 
mfoi for nearly every dump, ftui in the case of a modern Le 
nember of dumps, with an in eaeh, isquaTA should do (juile 
well. {See $12,5."} Generally, we would Wiped rt; RC E PTRQN to be 
slight ly belter ibjrt SAVES because it exploits behavioral feedback, 
worse because of Undue Sensitivity lo isolated cuorS, 



Figure 12 0 

One w ould expeel the nearest neicmesgr procedure (0 do wdl 
under a eery wide range of conditions-. Indeed. NEAREST 
NfciCillbOR in Ihe limiting case of recording all <$'$ with their class 
names, will do sii least as well as'any other procedure, There are 
condition s. though, in which n earest NPIPHftOFt does not do so 
well until Ihe sample siye is nearly the whole sprite. Consider, 
for example, a ^puce in which (here are [wo regions: 
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In iIli? upper FL'jj.kni a fraction p of ihc points arc in F+, and these 
;iri; nindoiTtly distrihuicd in space. and similarly for f. in the 
lower region. Then if a small fraction of the points are already 
recorded, (lie probability that a randomly selected point hits the 
iiline F Lis ili fli^rcst recorded neighbor is 

P' + y* - j - 2j»*> 

while (he probabiliiy i>T cor red identification by JtoVES or by 
best plane is simply p. Assuming that p > •] (if not, just su¬ 
ch ange p and q) we see that 

F.rrL>f taST Ma ki: < HfruTm ^ t[ir kcicuwk ^ 2 k Ei ror,, st p| J l :c 

$0 that neabfst keiciiiuir is worse than best plaNB, but not 
arbitrarily worse Thij effect wall remain until w many point* 
have h«n sampled that there is a iubstSfllial eliariee that (he 
sampled point has been sampled before, that fs, unlil a good 
fraction of the whole space is covered. 

On (he other sidd (o the client that (he “wiaitlj," of F + and 
F it less severe isce Figure li.T), Ihc nearest neighbor will 
cor verge to very good scores as SOOfl US there is a substantial 
chance of finding cme sampled point in most of the mkroclumps. 
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A very bad ease is a parity I ike structure in which NUarlsT 
Nt'ltitluOB rtclufillv does worse than chance. Suppose Lhat 4 t Fi if 
and only if v, v. ] for an even number of r’s. i'hcit, if there 
are jt ^"s,cach 4 will have exactly n neighbors whose distance d 
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satisfies G < J < V. Suppose 1 h4iL Jill bill 3 fraction of aft pos¬ 
sible Ifr's have already been Seen Then nearest keighsur will 
*!fi ors a given # if it has no! been wen {probability - blit 
one of iis immediate neighbors ha* been seen {probability m 
I — i$'\ So the probability of error is £g(1 “ f 1 '), which, For 
large n. cm be quite near certainty, 

This ciaoipk is, of course, ' L pnLhologicah" its mntbcrrtiilteiaus like 
Lo soy, and kesk. r k f NEtctihOft is probably good in many refit 
situations. Eli performance depends, of course, on the precise 
'’metric" used lo compute distance, and much of classical statist 
Lieut techniqiw: is GOflMftkd *Uh optimising coordinate as.es and 
measurement scutes for rdiried applications. 

Finally, we remark ibai because the memory and compulation 
costs for this procedure arc SO high., it is subject to competition 
from more elaborate schemes outside the regime of linear separa¬ 
tion -and hence outside Lh e scope oF lh is book. 

12.J Decisions Eased iml rrnhabillllca of Predicate-Values 
Some oT the procedures discussed in previous seel ions might he 
cnJiled "statistical'’ in Iht weak sense that their success is no! 
guaranteeil except up to some probability. The procedures dis¬ 
cussed in litis section are siulisticai also in the firmer sense (hat 
they do not store members of the tfsrre swr direelly, bgl instead 
store stultslicrtl furuiiHlers. Of measurements, about Lhe data ict. 
We sh il l I analyse i p deco il a sy Hiem that enm puics- or csl i males— 
Lhe condilionul probabilities^ thaj‘, for each class F^the predicate 
yr, has the value i It stores these pj s- together with the absolule 
preiLnibi I il ies p, of + bd Itg I n eoch of the F/s. 

Given an observed ■$■, lhe decision to choose an F, is a typical 
stJlistie.il problem usually solved by a '‘minimum likelihood" Of 
Hayes-rule method. It is iulcrcSting that procedures oT this kind 
resemble very closely the percept run Separation methods. In fact, 
when we can assume that the conditional probabililies p,, arc suit¬ 
ably independent {§12.4 2} it Hi mi out thdl the bcvl procedure is 
the linear threshold deeisiem we exiled bay it? In §| 2,2,2, W( now 
show how ’his comes about. 

I 2.4. L M ,|\iiiinm 1 il. e lib anil and BjycS Law 

In Chapter I I ■■■ e assumed that each <t> Is associated with a unique 
F We now consider the slightly more general case in whkh the 
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sj- ini; t eould be prodtwed by evenli in several different F-classes. 
Then, given an observed $ we ennnot in sercra! he sure which 
Fj. is responsible, bnt wecai at best know the associated probabil¬ 
ities. 

Suppose that a particular 4> L - Vtuis occurred and we want to know 
which F is most likely. Mow if is responsible, then the “join! 
c-vent” F t A has occurred; this has (by deli nil ion) probability 
{p 11". A 'h 5 )- Now (by definition of conditional probability) we can 
write 


<P(F,A*o) -tf^FJ-TOblF,). 


(■) 


That is, cIil 1 - probability dial boLh F^nnd ■ho will happen together is 
ci|ual ll> (he probability Ihal FyWiii occur multiplied by the prob¬ 
ability LhiLC ifFjiWlifl JO will 'ha. 

We should chonstf lhai F/uhich gives Formula I ibe largest value 
because that choice corresponds to the most likely of (how event* 
Iheil could have occurred; 

F,A** FjA*# ■” F* A *n- 

TLiesc are serious procticril obslade* (0 the direct use of formula 1. 
If [here are many different it becomes impractical to store all 
the decisions in memory, fei alone 10 estimate them all on the 
basis of empirical observation. Mur has the system any ability 
(0 guess about *'s il has not seen before. We ean escape all these 
difficulties by milking one critical assumplion—in effect, assum¬ 
ing, the situation closely fits a Certain model—that the partial 
predicates of =■ ... , e-„) art suitably independent. 

12.4.2 Independence 

Up nv navi- wcTimvc suppressed the Jfi orearlier chapters because wt did 
noi care whereche values of she e’s came from. We hring them hack for 
a moment hi that me can give a naLural eonteel to the independence 
hypsrllwfsis. 

We car evade the problems mentioned above if we can assume 
Lhitl the lest* i/rjI'A'i are istatisliciilly independent over each F^elasS. 
Precisely, Lhis means that for any H K2f) - 
ear assert that, for each /, 


W I V,\ - fP(tp, 3 F,} m : ■ ■ k (P <*. I FA 


(2) 
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VVe qrnpltuxire ih;it 1 his is a iwost stringent onnctiLicvn. Fox exam¬ 
ple, it is cquivalent Lo the asKilion that: 

CVi ffl that o t is in a certa ! n F -class, if one is told also the wines 
of some of liter's, this gives absolutely no further Information about 
the values of the remaining ip"s. 

Experimentally, one would expect eo find independence when the 
variations, in the values of ^'s are due to “noise 1 ’ ox measurement 
uncertainties within Ike individual median is ms (Figure 12.8). 



Ftj.ii r« 1I.S 




eptUGCWq; 


For, to the extent that these have separate causes, One would not 
eSpeet the values of one to help predict Che values of a not her. But 
if Lhe variations in the y^s are doe rn sefetiiatt of different X'$ from 
the same one would not ordinarily assume independence, 

since the value of each & tells us something about which X im F 
has occurred, and hence should help at least partly to predict the 
values of other *?"s (sec Figure 12.9). 



Figure 119 


An extreme example or nonindepcndcncc is the following: there- 
are 1 wo classes, F, and F\, and (wo p's, defined by 



a pure random variable with = I) ■ |. 

Its value is determined by tossing a coin-, not bv X. ' 

Wm ** f,> 

|l - ^ < V > if r r Fi. 
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Then CFCiPi A n|F,] * i 


[iui - I* i, 

N«(ife. ih-iLL withfr *?, nor *-j almt gfrvs my iitfortuaiSoft 

trfntirver ttfnwf Ff Each uppciirs (u Tjc a random coin toss, But 

from both Orte enn determine pvt (belly whkfc F has occttrtd, 

for if, " implies F h 

wtiile^n r* -fi implksF; | 

with absolute certainty, 

■rfmask,: We assure only indepirdeneo within each class Fj. Sc if X is 
noi go-en. shen learning one ^-vnlue rorr help predict another. For tr¬ 
ample, suppose Lh ;iL 



These two ^ “s are in fact independent on each F. But if v/edidnoi know in 
iutktitict >h<t\ X t F, hut were totd Lhnl «*, - 0, w? could indeed ilien 
prcdiei EtiaE if; * 0 also, without Lhu violating o u t Independence as¬ 
sumption. ill we had previously been told LbaL X t 1 1 . I km we foul'd 
.j.Vlvj./i' pre:! ie* the value at iii ;Ii:i 1 ease learning the value of -p \ 
would have no efl eet on fmr prediction nf ^2 ) 

12 4,1 Tiro nui xspieuhh IJUdJluiod decision, IW independent ^’s, is a 
liiLL'ar llireshnld predleale! 

Assume that the p 's are sloitstically independent for each K.. 


Define 


Pt - if (F,), 

Pti - &{<£* - I'ifjrK 

qi f - l - K “4T(iPi - ® |Fj)- 

?! expose ilial see have just observed a * = for*. Ph). a id we 

want to know which F, was most probably responsible for this 
Then. according <0 Formulas I and 2, »c will choow a ih lU j which 
masiniizes 


Pt ■ J { P<) 11 9v 
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DecaUSe Plains are me re familiar 10 ns than products, we will 
replace Ihcse by (heir logarithms. Since Hog x increases when x 
does, we still will select the latest of 



O) 


Because the fijthL-lturtd expression ssa constant (hat depends only 
Upon /, mid not upon the experimental ■>, all of Formula J can 
he written simply as 



O') 


Example I In Lho cose lhat ihcrc ^tc just two classes, F| and Fj, 
we can decide that ,V f F whenever 

5 + tfi, 

Lti :le is r when 


5C^rJ - 


0) 


v, hich has the form of a linear threshold predicate 
± HI. 

Thus we have the reinark;■ hle result, that Lhe hypothesis of inde¬ 
pendence among the leads diroelly to the familiar linear 
decision policy. 

F sample 2 (probabilities of error}- Seppo&4 that for all f, p„ - 
Then p r! is the probability (hat (A") And is the prob¬ 
ability that ^ £(.V), that is, that ev makes an error in 

(.individually) predicting the value of <f ■» \X * F|l. 
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Then inequality 4 becomes 



O > losj- 


t*') 


Now (bat (be (2*, - 111 lem has the effect of <mW^£ cr 

zubirnifitix iv r j according to wTielhur J 1 or 0. Thut, wc can 
think of the h' 1 * as weights to be added (according 10 the /'*) to 
one side Or (he other of a balance: 



The log {prfpi) is ihe -t ii priori weight" in favor oF F, at tLe 
Start, and each m „ - log^u/d.i) is the "weight of (be evidence 1 " 
that v 1 , = ( gives in favor ofF|- 

H w quite rcniiifLahb: that the optimal separation algorithm given Lhat 
the j-prnh.ihill Iks :i■ c independent bat the form (inequality 41 *T a 
linear threshold predicate. Dili one must he sure Lo under slimd that ii 
[ Vf,,* > hi is Ihe “niHiffia! 1 ” prcdwnle obtained fay the independent- 
prohuhility method. ycL does tm pcrretkly renlire a predicate i>. Itil* 
does not preclude lh« csi&Lcnec of a precise reparation [£*'**? > fl r | 
Uihieh.fl|wnyi jigruvs ^ilH t. flllis. is the siHmtdnn suited by Figure 
i ; 31 a>.| For inequality 4 is “uptirsud*’ only in relation to all Aj k pro¬ 
cedures rfeilT Wf na inJtffmtsSuM Stkit tbfW ihe coniiiliiifUfS pfobilfi/httt.i 
fp | add | y> .(. while a perccpiTuri coinpulei coeilicianLi-tiy -a non s-l-n^F-liic^l 
jisaich procedure ih-it is sensitive to individual events. 

Thus, if ^ is. in Fucl in /.(41 the pereepiron will eventually perform at 
least Its well ll* si ay lincar-sruLisEhcal machine. The lULtuf lai'tily enn have 
Lhc adv.inlnpe in some coses: 

t. ff i f.(*J Lbu statistical scheme may produce a good approximate 
cr/piiToLion « h ilc the- pcKVpl ron in ight fluctuate wildly. 

2. The link n> achieve a useful level may he long for Lhe pEiecpiron flic 
algorithm which is basically n serial search procedure. The llneaf-ilatis- 
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Heal mitchiitc is task-ally more parallel,, because it finds twh coefficient 
Independently of ill* oilers. and needs only a fair sample oF ihe f"s, 
{While superficially pcrceplron coellieienli appear lo be changed indi¬ 
vidually, each dcuiioi :i'nc-ui u change depends upon a test involving all 
Lkc L'Of ificieMfl. ) 


12,4,4 l-ayer^M iChirtW 

Forlrtitb 3' iugjesli the design of a spachip* Tof making $ur 
ckeislofl: 



D is a device ih;H simply decides which ot its inpuls is’the largest, 
Each ^-device emil* a iiartdnrd-siicd pulse |if - I] wheit X 
is presented. The pulses arc mull ip lied by the »■„ f|uanlilieS at 
in dialled,. itnd summed at the E-box-c*. The ft, terms may be. re-- 
garded its corrections fur Lbe extent Hj which the p„ H s devim-e from 
a central value of J, combined with the a priori bias concerning 
¥ t il self. 

It is often desirable to minimis the etuis of errors, rather than simply 
:li; chance of-error. I f we ddliit u> he Lhc coat of guessing Fj when il 
is really F, Hut has occurred. Ihen il is easy to shew that Formulas ] and 
2 now lend lo rinding Lite k (hat minimises 



where S s = II ll is inleresLing ihai Ibis more complicHl-td procedure 
also lends itself to the iiiii1rila>er structure. 
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! 



It Lu be possible Id devise u Jiining nljorillun to optimize the 

wsijfcL* in Lhiis using. «y, she nuigyiitude of a reinfcr^mcnL signal to 
rammunictife 1* lilt net the rod uf an error. We have not investigated 
Ih-is. 

12,4.$ Probahllily-estfimition procedures 

The Ant algorithm for ihe haves linear-statistical procedure has 
|i> compute. Or estimate. either (he probabilities and p, of 
formula 3 or other staiislkal quantities such as Ll weifiht of evi¬ 
dence" ratios p/{ 1 - p). Normally these cannot be caEctiiuled 
directly {because they arc. by definition timits) Sb one must find 
L r sritiwtvr,<;, The simplest way LO estimate a probability is to 
find the ratio ff/N of the number U of "favorable" to the 

number ,V of all events in a sequence. If ^ l ' 1 is (he valoe of p on 
the d ill trial, then an estimate offPte - 11 after n trials can l?c 
found by ihe procedure 


s fAnT: 


sstfat: 


Set a to U. 
&ct n to 1. 


Set«10 


(ri - l)u 4 if 




SctnlQH + I. 

frO H> B EPF At. 


which car easily be seen to recompute the ""score HfN after 
each event. 

This procedure bus Ihe (lis:idvuntrt(C that : t has to- keep a record 
of n, lilt number of trials. Since n increases beyond bound, this, 
would requite unlimited memory. To avoid this, we rewrite the 
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a hove: program’s com pul ution Ln the form 



This. suggesLs a simpler heuristic substiluic: define 

T-">". ’ j 

\ * W - (1 - ^L'"1 + * ♦ # W , 1 (S> 

where-e is a "’small” number between 0 and I. It is easy Co show 
Lit;i[ as rj increases Ihe expected or mean value of or w h which we 
will write as {nr’" 1 }, approachesp {that is, ^ }asa limit. For 

^« 1 '^ - (I “ iKra iJ ">+ £^'9 ™ ip 

- M - fl - t>]* 

And 

- 0 - 00 - {1 - r)V + *P 

* Cl - (I - 0V 

and GO* Cart verify that. for all n, 

<w IH > “ Cl - (I - <Y)p 

~ ‘ p, (as jt —» ^ ) 


Thus. prc?oess 5 gives art cslimaiion of the probubiliiy that ^ = I. 
A metre detailed analysis would -show how (he estimate depends 
on Ihe events of the reCenl pnsl, with the effect of ancient events 
decaying exponentially with coefficients like (I - 

ResMuse process 5 “forgets.” it certainly does not make “oplimal” 
use of it* pa st experience; bul under some circumstances it will he 
able In “adapt" to changing environmental statistics, which could 
be a good Ihing. As a direct con sequence of the decay, our esSi- 
mator has □ peculiar properly: its variance "V 3 "' does nOl Sp* 
preach cero. In fact, one cart show that, for process 5, 




oP 


— pi i - p) 
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and tli ps, while not icro, will be very small whenever r ia. The 
sil Million is thus quite differem (rum She HfN eslimate—w^josc 
Vniriquce is p (I “ p)fn and approaches 2 ero as rt grows. 

In fuel, we can use the variance to compare she two procedures:: 
If we "’equate" She variances 

pH - p) ‘ ~ pH - p) * v- 

d! — E rT 


obl-aiit 



t 


suggesting Shjl the reliability of the estimate of p given by process 
? is about I he same as we would get by simple averaging of the 
hist 2ft samples; thus one ean think of the number 1 ft as cor¬ 
responding to a "tinie-eonstanf" feu forgetting. 

Another estimation procedure one might consider is.: 


I urltALI^t: Set * to anything. 

HtVK^r: If ^ rn S, Sit a to a + 1 , 

rfiP *■ C, Set HT tL> (t -r [)£T, 

Go to he peat,_ 




CoitirerevncelotlK Fi.*cd-Paini 
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L>r, equivalenll), one could v. rite 

- {1 - 4 0 + c a >- l V W - 


I|«n be shown that this has an expected value, in the limit, of 



tl is interesting that a direct estimate of ihe likelihood ratio is 
obtained by such a simple prOCCW as if ^ ■ L odd 3, orfierwrj* 
mfitrply A.r [I - *). The variance, in case anyone cares, is 


1 



3 - <1 ' 


11.4.4 The Samuel compromise 

In his classical paper about “Some Studies in Machine [-earning 
using the Game of Checker*" Arthur L Samuel use* an in¬ 
genious combination of probability estimation methods, in his 
implication il occasionally happens ihnl a new term 

ittiTif-JuCi'tt [and an old one is abandoned because il has not been 
of touch value in the decision process}. When this happens there is 
a problem nTpreventing violent fluctuations, because after one or 
3 few trials Ihe term's. probability estimulc will have a large 
variance as compared wish older terms that have belief statistical 
records. Samuel uses (lie following algorithm to “stiibili/i -1 his 
systent: he sets a 101 to J and uses 



where N is scl according to the ’‘schedule' 1 : 



Thus, in the beginning she estimate is made as lhough Ihe prob- 
ability butt -ilreudy been vxiiiu;iteJ to be 1 on the basis of several. 
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I lint is, the order nT 16, trials. Thun In the middle" peri m3, the 
algorithm approximates die uniform weighting procedure, finally 
(when* - 256) the procedure change* to |he «^.ja»ncnLi»t decay 
mode, with lined A r , so 1 hat rceent experience tart OULweigh curlier 
results. {The use of the powers of two represent* a e&ns'en left! 
COmputer-program technique for doing this.) 

In Samuel's system, the lei ms aetuaELy used have the form we 
found in inequality 4'of §12,4.3 

2** 1 - 1 

so Lhat the "estimator" ranges in the interval -1 < p (, l < +■ 1 
and earn he treated as a “correlation coefficient," 

12.4.7 A sintptr “synaptic** rrinforrer theory 

Let US mitten simple neuronal model." The model is to estimate 
pu - F,L WS»ng Only in format ion about occurrences of 

Ivb - 11 and of f# r F.1. Our in odd t^ill have the following 
‘‘anatomy": 




The bug B, contains a very high and constant concentration of a 
substance E. When ^ or Fj occur or ”ftrd'—(Ike walls of the 
corresponding bogs B, .and/or C, become '‘permeable*’ to £ for a 
moment, ir^. alone occurs, nothing really changes, because ffj is 
surrounded by the: impermeable Cj . If F, alone occurs, Cj loses 
some E bj ililfusion to the otnsidc: in fact, if o is the amount of 
E in Cj it may be assumed {by the usual 1 laws of diffusion and 
concent ration} to lonesome fraction * of «; 



tf'fiuifli yr, nud F, occur then upprosiniitely the same loss will occur 
from c r Sin*ultaneously, :m. esseiitially coustaiii amouni k will be 
1 'injected" by diffusion from B, to C s . 5c 
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(We can assume that f> is constant because ihe concentration of E 
is very high in B, compared to that in Cp One can invent any 
number of similar variations,} In either ate we get 

- (1 - <)ct + ct 

SO that in the Ei mil the mean of o approaches - p (as can be 
seen from llie analysis in g32.■*«5>. This is proportional to, and 
hence an estimator ofp,* - f Fj}, 

Thus the simple an atomy, combined with the membrane be¬ 
coming permeable briefly following a nerve impulse, cool'd give a 
ynanLijty that ss an estimator of the appropriate probability. 

flow could this representation of probability be translated into a 
useful neuronal mechanism? One could imagine all sorts of 
schemes: tonic concentrations--or rather, their logarithms!— 
could become membrane potentials, or conductivities, or even, 
probabilities of occurrences of other ehemfeal events. The ra¬ 
nsom y“ and '“physiology’ 1 of out mode! could easily be modified 
So obi aim likelihood ratios, indeed, it is so easy to imagine 
variants— Lhe idea is so insensitive to details—Lhat we don’t pro¬ 
pose it seriously, eaccpL as a family of simple yet intriguing 
models Llki t a neural theorist should know about. 

12.5 Ajk nlguritlLftis fdr the I SOU .47 A pmfeduie 

In this sod ion we describe a procedure proposed by G. Ball and 
f> Hall to delineate ”el osiers” in an inhomogeneous distribution 
of vectors. The ideals best shown h> a pictorial example, imagine 
a two-dimensional set of points |*| that fait into obvious dusters. 
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Be.ein by placing a few "elustcr-poinls" A‘, |; imo the space at some 
arbitrary locutions, say, near the center. We (lien divide the set 
of *’s into subsets H ft .assrgrrirtg each $ to the Jjorrmrl Al 11 point; 

i 





Nest, we replace each A I 11 by a new cluster-point Ap 1 which Is (He 
nrcvproF ccnlcr-of-gravtty or tire +’s in S„ and then define Hp 1 to 
be the set of +’s nearest to A?: 
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Repeating, fce get a netv set oF A,’s and ftjV 




Prom now Lin, there is lilLle or no change the duster-points hnv* 
'■fontid the dusters,” 

Stall nnrt (Call give j number of heuristic refinements fur creating and 
destroying adriiLtonal eJnslcr jroinlsj Fur sample, to add one if fh^ 
variane* of an R-set is “too large” and 1* remove one iF L*o are “too 
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close"' logether. Of course, one csn usually “see’ 1 iwo-dimensioual 
clusters by iasjwclicm, bul (SODaT* u allujjEiJ tn glv E useful results kit 
fl-dtmenEioasI problems where “insped ion"" is uni of Lhe question. 

To use (his procedure. ifl ou r eonlent. \vc need some way to com* 
bm? its automatic dtUtsUtcation ability with the information about 
the F-dtlsscS. Art obvious first sicp would bo to apply it separately 
to each K-cluSF. and assign all the resulting A’s to that class. We 
donoL know mudi about ip ore sophisticated schemes that might 
Seud to better results ifl Lho A fnJ stage. 

!2.5,1 An IMEHT,* convergence theorem 

Therein n iheuncrn aboui jsooa.ta (sold u> us by T. Cover) that suggests 
c3m| ii leads to some sort «r local minimum. Ln us formalize the pro¬ 
cedure by defining _ _ 

A" 1 ;-#) - the A'^rM is newest tp +, 

{If there aie several equidistant nearest A/s, choose the one with the 
smallest rude*.} 

R 1 "' - the sd of iS’s for which A 1 ’* ♦ — A^. 

A 1 ” 11 - mean (■'ft 

Finally define a. ■‘Score”’ 


^ - Ei* - 





j' i 



because I be mean (A 1 , 1 " 1 ') of arty set of vectors |R?| is just Lise point 
that raiaiuvtfos the squared-distance sum to all the points (of R^j. And 
this is. in turn, 


iEZi-f at"! 


* _ *>♦’! 


1 Hit* 'F 
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because cadi point win be 1 ranskrrwf (0- an ft 1 ?" 1 ’ fu/ *hirfi the dLHance 
i: : - 111 ImrYI al, I h ;■: is, fi^r I|Ji. 



0 limit. If theft if OnJy tt finite ttfl of ^'.t, the A‘r mux! flop (hanging in 4 
fi> lire tmitber of steps. 

For in the finite raw Ihsre ate only a Unite rtunber of partiiicmj gR.) 
risible. 

E2-?.2 ■ ncT-rm cn r:Lt mcrlm ds 

In analogy In tin '"reinforcement" methods in §12.4,5 we can ap¬ 
proximate m.tubythe Following program: 

st x mi ChOtWc a set of starting points A b 

BFPEAT: ChOOSe S -t. 


Find A{$): the A, nearest to 

B ,<L I'Ji'i V.,. J'l _ /lii'.t'i L , . ,* 



It is cleat that this program will lead 10 ^nalita Lively the same sort 
□ Fbehavinr’ the A.’s will tend toward the nm™ of their H : regions. 
lint, Just as in §12.4, the process will retain 4 permanent sampling- 
and-forgetting variance, tvith similar advantages and disad¬ 
vantages. In fuel, all she A flfc slgoriEhm* we have seen can be so 
approximated there always seeing in be a range from very local, 
incremental method*, to IflOfC accurate, Somewhat less "adap¬ 
tive” global schemes. We resume this discussion in §32.8. 

12.6 "["[me rj L Memory for F,\*et Watching 

Suppose lhat sve m* given a body of information— we scil] cjl| 
it ihe ditto sti in the form of 2* binary words eaeh h digits irv 
length (Figure I2.ICH; one Cnn think of them as 2° points chosen 
■it random from a space of ?* points. (Tale a million 2 r 2 a word* 
of kuglh 100, Tor a practical crumple.) Wc will suppose that (he 
data set is In h< clu>.svn 41 fSUidorfi from all possible sets so that 
one cannot expect to find much redundant structure within it. 
Then the ordered data set requires jibovC h ■ 2^ bits of binary 
urformal tan for complete description. Wc won 1 !, however* be in- 


-Vl ■ 
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(crested in Ihe order of (he words irt Ehe daLa set. This reduces 
Ihe amount of information requited CO store the set to about 
(ft - a) - 2 ■'bits. 

W't want a machine Ihtil. when given a random fi-digii word w, 
will answer 

OL-es tion 1. k_w ill i h<; dal it sct? J 

and wc want to formulate constrain Lg upon how this rttachine 
wnrlss in such a way llial we can separate conipuiatiOrtal aspects 
from memory aspects. The following scheme ildlisvts this goal 
well enough to show, by examples, how link is known about the 
conjugacy between lime and imemory. 

iVr will pivc our machine a meriorj of hf separate bits- that is, 
one-digit binary words. We art required to compose in advance, 
before we see the data set two algorithms A fJt and that 
Satisfy the follow mg conditions: 

E. A-ti, is given die d.iia set, Using mis as data, it fills the M bits of 
memory wiih inforimiLinn, Ncudierlhc dn1a set nor A|.« arc used again, 
nor it A irj allowed to gel ;iny information abcul wfiJE Ahk did, CKCepL by 
insfveccing Lhe eonien is o f .if. 


l Wc u.ill git EoQuertlen Jin uhutiE fifltcnm mules. 
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2 Lhcn given a randnni w«rd, w, anti asked Ed answer Question I, 

■>n r|i. like irfi'rnilipn slttrci in lire memory by A We are iulsrcrted. 
in how T-nny bil* A flnd has (qc-puiall in (lie prujcei?. 

3. The J..UI is It? t>pEtini?c (he design of A fllt and A me (0 minim ir« 
ihr numhqT t?F memory references in (he qiieiiicm-jins^crine compute 
Lion, averaged over ;ilL possible words *■ 

IZ.6.1 t'nsc li Fnoriiioiis Memory 

It is plausible Ih:it the larger be M, the smaller will lje the average 
number aF memory-references A r ** must mate. Suppose that 

Af > 2 \ 

Let mi be I he tlh bn in memory; then there is a bit for each 
possible query word m 1 , and «c can define 

{ A* k : Set ht^Id I if w is in the do La set 
<i 4 ndi *■ is in (he Onl j set i f m „ - L 

Thus. with a huge enough memory, only one reference is required 
to answer Question I. 

CI.G.2C use 2: Inadequate Memory 
Suppose ihal 

M <{b - a)2 a . 

Me re. the problem cannot be solved at all, since cannot store 
enough information Lo describe the data set in sufficient detail. 

■TilSr .3: Binary i i^antlimle fvurl 
Suppose thai 

Sf - i> V. 

Now there is enough room lo store (he ordered Jala set. E>eRjne 

sinre the wprds of the data set in ascending numerical 
order. 

A rt ^: perform a binary search to see first which half of memory 
_ might contain w, then which quartik, etc. 




r\ 

i 
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This will require at most a * log 2 tf insp.celLcuts of i-biE words, 
ihi;i e Ls, o - b bit-inspections. 

Thsi ii not an nplimnl search since, (I) tine decs rwl llwayi iiecd! Id 
in-jpL'cI a 1 1 1 . 1 !i■ i'l- ■ ward Id deride which Yvurd Lli imjxcL ceil, arid (3] il 
dues ihtI e-splnil Ihc uniform it y of distribution ihal llic first a digils of 
I ho iifiifrfvi da I a set will (on Itic average] show. EITecI L reduces (he 
required number from a • 1^> | h e order of |j o - k and clfc-eL 2 tcducil 

il from a ■ b Ih> a ■ ih — rr). We don’t know cruelly hnw Ikes: Iwo 

elf«t* com hire, 

11.6.4 Case4: Ethaustiao Search 

Consider 

.if - (b - o)l*. 

TliLs gives juse about enough riteiJlOfy 10represent the unordered 
da(a set. For example wo com Id define 

\ nM : First put the wards of the data ME in numerical order. 

Their com p ule iheir siteefrssi-vJi difftreneei. These will re¬ 
quire ubout (i - a) bit* Meh. Ust a standard method of 
information Lheory. Huffman Coding (say), to rep resent 
(his sequence; it witl require it bout f tr - a) 2' hits. 

13 the only retrieval schemes we can think of are like 

A i, 4 \ Add up successive drlfcrcn-ccs in memory unlit the surrt 
equals or exceeds w. If equu.fi Ly Occurs* (ben to is in (he 
data set. 


A ml i hi* requires - - a)!’ memory references, on (he aver¬ 

age. Tt seems clear (bat, g,iven (his limit on memory. hO Am - 
A^j pair ean do much better. Thai is, we suspect that 

If no extra memory h iPwitaMe ihm. io ttQmfttifrt I. Ont 
ttiNM, . or lire fjrc/YJjr l M arch through half ihi* muntary. 

One might go sbphi]y furlher uvea HufTrrNin Coding needs some 
euro merh Pry. and if I here is none, A* fc can only siore a a efficient 
■ L rnji*lber"‘‘for ilm whole data scE, Then the conjecture is that A 1rJ 
itiu.ii _i hii i.'iv, i always look, at aimosl all of memory. 
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12.4.5 Case 5; Hush reding 
Consider 

M =. b - 2 * ■ 2 . 

Here we have a cafe in which there is a substantial margin of 
extra mem cry--about twice what is necessary for the dale set. 
The result here is really remarkable —tuie might even say eounter- 
Intuitive -because the mean number of references becomes very 
small. The proccdarc uses a concept well known to programmers 
who use il in "symbolic assembly programs” for symbol-ln ble 
references, but does nut seem to be widely known to other corm 
pule; "specialists." It is called hash-coding. 

There are man} variants of ibis idea W? discuss n particular form 
adiipitii Lu a redundancy of two. 

In the hash-coding procedure, Ap* is equipped with a Subprogrant 
R{n\J) that, given an integer/ and a fr-bil word w, produces an 
(j? + tJ^b-il word, f'hc function fl(uyf) is "pseudorandom rl in Lhe 
schso (hst Tot Csicll /, R(™ t j} maps the Set of nil 2* inpul words 
with uniTorm density on the 2 W *' possible output *ordi and,, for 
different j\ these mappings are reasonably independent or 
orthogonal. One could use symmetric function*, modular 
arithmetics, or any of Lhe conventional pseudorandom methods.* 

Now. we think oTthei j 2 3+ -bit memory as organized iplo fc-bit 
regislers with {a -i- U-bil addresses: Suppose that A fit has already 
Hied the words . . . and it is about to dispose of w H +k . 

A k.: Compute A (w*, t , 1). Ef the register with this address is 
empty put h'jti in it. IT that register is occupied do the 
same with 2), d),... until an unoccupied 

register k , is funnel; ftle w„ + l therein. 

A(, r y. Compute fljw. I). If this register r&ttlaim v, that w is in 
1 he data (hit. 1 f R I ) is ein ply,. then w if tint irt the data 
set If At"., i) eon tains some other word not w„ then do 


'Thfra-kf u iti;iL rc-qwinij.Hinc inap'cul [ireip^rLy Ilia! can only 

N .'.ppriwim.iud. h is true Itint any p.iiiiculiit ft wilt |%* hml on ptrthvthtr tUn 
jtls, hill [here is no pn -htan nt alt - Si.n we cnntiricF nverii|je huhivinr on all 
[KeuhLc drill pels. 
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(he same with 2), and if necessary i?{irr, 3). fl(w K 4), 
-... until cithfcf w or an empLy register is discovered, 

On the arerajte. A b , : nitl wake less than 2b metnary-Mt references] 
To sec this. we note firs! that, on Lhc average, this procedure leads 
(ci the inspection ofjsi-EL 2 registers! For half cif (he registers are 
empty, and (he successive values of /?(ivj) for j = 1.2, ... are- 
independent (wilh respect to ihe ensemble of all d-uta sets) so the 
mean number of registers inspect Id fin Jan empty register is 2. 

A-uiuati}', ihe mu.in termination lime is slightly Its. t, because- for w'i in 
[he d.-iEa srf (he e*pee(«f number of inspected registers is < J. The 
procedure is useful R?r symbol-tables and (he like, where otic may -want 
nut only to Ln iF n is there, but also to retrieve some other data as- 
snriiued (perhaps agii in by h asb-eoding) with it. 

When [lie margin of rciturtdii,ncy is narrowed, for example* if 

M - b - 2\ 

n - I 

then only (I/nth) of (he cells will be empty and one can expect Eo 
have to inspect about n register*. 

Decause people are accustomed to the fact Ihal most computer* 
tire “word-oriented " and normally do inspect b fail* m each 
memory cycle the following analysis has noi {(0 our knowledge) 
been carried through fo the case of |-bil words. When w<t pro¬ 
gram (o match words f>K by bit we find that„ since half 
(he words in memory arc rcro, matching can be speeded up by- 
assigning a special "zero" bit to each word. 

Assume, for Ihe moment. (bat *e have room for these 2* extra 
bits. Now suppose ehal a certain m,-<j is not in ihc data set, (This 
has probability 1-2"*.) First inspect (he “/.sw"' bif associated 
m hh fl(w^l). This has probability i of being zero. If it is not saa, 
then we match (he bits of w„ with those of the word associated 
wilh Jf(wg.L). These cannot all march (since n‘ D isn't in the dal a 
set) and in fact the mismatch will be found in fan average of) 
2 =* f + J + 4 + ■ ■ - references. Then the "zero’" bit -of Afwn,?) 
must be inspected, and (he process repeals. Each repeat has prob¬ 
ability | and Lhe process terminates when the “zero’ 1 bit of some 
- Q. The expected number of reference* can be cminted 
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then as 

1(1 + 2 + i { I + 1 4 i . r)B 4 I -5+1 - 4, 

](Tiy, i.v in Jala set |:i il event whnu probability is 2 ,_l ) and we 
repciil I he analysis we gel 4 + references, because the process 
musi lerirHiMK by matching aLI b hits of wp, 

The expected number of references, overall, is then 

4(1 - 2 s "*) + (4 + b)2- b -44 A > V~ % 

- 4 

since normally 2* '* wi]] be quilt Liny. We consider It quite re* 
markable chat so SlLtle redundancy—* factor of two—yield* this 
small number! 

TJie estimules above are on Ihe high side because in lhe case that ms is 
fir the da La set the ‘'run tenglh" through will be shorter. by 

nearly a Factor or 2, than chance would have it just because they were pul 
there by A h *. Orv Lhe other hand, we must pay for Hit exlra "zero" 
bits we adjoined to M, Ef we have W - 2b-Z“ bit* and make words 
of length & + I imsienr! oF b, then the memory becomes sli^hily mart 
thiin halT Full: in fad. we must replace ""4” in cur result hy something 
I ill e i |<> + i)/(fr - I )J. Perhaps these twn effect* will offset drie another; 
we haven't mmEe c*ne-1 mfeulfliigrij^ mainly because we aye hot *UTe that 
even this A fpi -A frrJ pair i* optimal, 

[1-curlninly scuir.s suspicion* Ilia! half lhe wuriSi in memory site limply 
amply! On lhe n'hcr hand, lhe besL one Could expect from Further i«i- 
prcv:.r,|! ibc algnriih-ms is to replace ^ by 3 (or 2?). and this I* not & Fargo 
enough: >m rrot Ip walk hjri for. 

1 1. ti.fi Summary of E strict Matching; ALgmrEtluns 
To summarize our results cm Question I wc have established 
Upper bounds for lhe Following oases: Wc believe that they arc 
close t5 lower hounds also bul, especially in oases 3 nnd A, arc 
not sure. 


Case 

Memory size 

Sit-references 

M elli od 

2 

<l (b - a)2 4 

DC 

(impossible) 

4 

f b - a) 2* 

\{b - a}2* 

(search nil memory) 

3 

b-2* 

4 i ■ rj 

(logarithmic sort) 

4 

lb ■ 2 fl 

4 + r 

(has-ti coding) 

l 

* 1 * 

1 

((.able look-up) 
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LI..? Time is. M t'liiur_L for Best Matching! An Open Pj+iIiI c-aii 
Wt bate stnnmariifed oar (Incited) uiw! crusading. tvF ’’Question I” 
— [he exact matching problem—by LSic- little (.able in §i2.b.fr. IF 
on< "plots the curve” one is instantly struct by the effectiveness 
oF small degrees of redundancy. We do not. believe that this 
should be Eaten Loo- seriously, for we suspect that when the prob¬ 
lem is slightly changed the result may be quite different, We con¬ 
sider now 

QUESTION 2i Given nr, exhibit the word wdo^st 10 w in the data 

set. 

The ground rules about Am, and are I be Same, and distance 
can be chosen to be (be usual metric, that is. (he number of dibits 
in which two words disagree. If jr : ,,.. ,* t amt if],.,.+ St are the 
(binary) coordinates of points «? and w 1 then we define the 
Ifijnwtiug distance to be 

' 

ki - il- 


One gets exactly the same situation with the usual Cartesian 
d islance C(h 1 , iv), because 

[£>■, flr)J> nr £ | Jf - ij 1 - 3 k - xA - 4 {v,w) 

so both C(n’, H'> and tl{*\ u 1 ) are minimised by (lie Surrtt w. 

11.7.1 C ase 1; M r* L 

Aw. assigns for every possible wprd * 4 block of b bits thai con¬ 
tain the appropriate bits or I he correct 
A i„t looks in the block for w and writes out w. It uses b references, 
which is obviously the smalLcsl possible number. 

12.7.2 Cat* 2: M < {b - n)2 4 . 

Impossible, for same reason as in Question I. 

12.7.3 rase J: M =. b‘2* 

No result known. 

12,?,+ Case 4s M - {b - all* 

This presumably requires [b - a)'2‘ references, ibal is, alt of 
memory, for Lhc same reason as in Question \. 
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12,7,5 Cuse 5: {b - a) 2* < M < h ■ 2* 

Ny useful results known. 

l2-7.fi Klunmy Praspicts for Best Matching Alger if lints 
The results in £12.6.b showed ihat relatively small factors of re¬ 
dundancy in memory size yield very large increases in speed, for 
serial computations requiring the discovery of exact matches. 
Thus, there is no great advantage irt USieg parallel computational 
mechanisms. Tn fact, as shown in §-l2.b,5 n a memory-size factor of 
just 2 is enough to reduce the mesa retrieval time to only slightly 
more Lhan the best possiblCr 

But, when we turn (0 the /*?.</ nrafeA problem t all this seems to 
evaporate, in fact, we conjecture that even for the b est possible 
Ap.iMinj pairs, the spKed' Up value of large memory r edundancies 
is very small, and for Tarji c 3S5 sets with long word lengths there 
are no practicj iTglternatives 10 large searches that insp c.'l large 
jia rts of the memory , 

We apologize for riot having a more precise statement or Ihe con¬ 
jecture, or good suggestions for how to prove it, for we Tee! (hat 
this is a fundamentally important point in (he theory of eompitta- 
lion, especially in clarifying the distinction between serial and 
parallel concepts. 

Our belief in this conjecture is based in pari on experience :in find¬ 
ing fallacies in schemes proposed for constructing, fast test- 
matching hie and retrieval algorithms, To illustrate this we discuss 
next the proposal most commonly advanced by students., 

12/7,7 Thu? Numerical-Order Scheme 

This p reposal is a natural attempt to extend the method of Case 3 
from exact match to best maich The scheme is 

A lih .: store the words of the dal a -Ml in numerical order, 

A rt „,: given a word w h find (by som e procedure) those words whose 
first a bits agree most closely with the first, a bits of w, Row 
to do this isn't flear, hut it occurs to one that (since this is 
the same problem on a smaller scale!) the procedure could 
be recursively defined. Then see how well the other bits of 
these words in atch w il h w. Nos t, 

The intuitive Idea is simple; the word w In the dal a set that is 
doml to u- ought to show belter-than-chance agreement In Lhe 
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first a bils, so- *hy not loot fia'si at words Itnowrs to have (his 
property. There are two disastrous hogs ip this program: 

]. When can one slop searching What should we fill ip where »e 
wrote "West . r r {?),. r We know no nontrivial rule that 
gnaranteei setting the best match. 

2. The intuitive concept, reasonable as it may appear, is not 
valid' Et isn't even of much use feu finding a good match, let alone 
finding (he fees! match. 

To elaborate on point 2, consider an example: let a - 20, b - IS.fiOO. 
Let *, for simplicity, he the all-zero word, A typical word in the data 
set will have a mean of 5000 wit's, and 5000 rrro’s. The standard 
deviation willte 10^000)50. Thus, ktss than one word in 2* - 1™ 
can be expected io have fewer than 4750 raid’s, Hence, the doseiL word 
in tine data set will (on the average! have at least this many one's. That 
closest weird will have ton llie average} >20-(4750/10,000) ** 9.J orw's 
among its first 20 bils! The probability that v will indeed have vfcry 
few flwr's in ii* OFil 20 hits is therefore eniremely low 1 , and the slight 
favorable bias obtained by inspecting f*ere words first is quite utterly 
negligible in reducing thq amount of inspection. Besides, objection 1 still 
remain;. 

The value of tf rdc?u:£ the first few bits of Lhc words is quits useless, 
Lher. Classifying words in Lbis way ucp runK in the ir-dimensional 
geometry, lu breaking bp die space InLu '“cylinders" which are not well 
shaped for finding nearby points. We have. therefore, tried various-ar¬ 
rangements Of Spheres. but the Siifle lr.uti of trouble appear (iflET mure 
analysis). In thu course of thaL analysis,, we are led IW SUSpcct IhaL Ihere 
is a fundamental properly of .i-dimcniiunall geometry lh.it puls a very 
strung and deseo u raging liiiiilalsun up.: 11 alt jilch ulguricSnnS. 

3 J.7.8 Why Is Best Mftfdi an IMHVrtnl from Exact Mulch? 

If our unproved conjecture is true, on* might wadi at least an 
intuitive explanation for (he cljlTefence we would gel between 
■JE2.6.3 and 12.7J. One way lo look ;K it is to emphasize that, 
though the phrase's "best match" and "exact match” sound simi¬ 
lar to the ear, they realty are very diirereni. For in the case of 
exact mutch, tn> error is allowed, and (his has the remarkable 
elfccl of changing an nSmensiaaoi pt-ahtetn into a Cntt-dimmsionaf 
prvtrfvn r. r For best matching we used the formula 

6 k 

=■ Ur - *ii “ 3 Ur - irL 

r -1 r-j 


Lrror 
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where we have inserted Lhc coefficient H i" to show that ail errors, 
in rfi(Tcrent dimensions,,-are couMed equally, I5ui for exact mutch, 
since no error can be Lolcioled, we don’t havti to weight them 
equally: any positive weights wilt do! So For enact match wc could 
just as well write 

* * 

Error = 2 .1 2 ‘ I*. - £J or even Error - ^ 2 '{jk, - Jtj) 

j-i r-i 

because c ither of t hese con be aero only when ^![ jt, t ^Shades 
of Stratification.) But then we can (finally) rewrite the hitter as 


Error - EE ?x t ) - (2 2'^) 

and we have mapped the n-dsmensiona] vector (jt|,. P ,, into- a 
sirglti point on a One-dimensional line. Thus these superficially 
similar problems have lolally dilTcrent mathematical persona li¬ 
lies! 

T2.Jt Lnrre-mmlui Ciirn[mtjriiVrt 

AH the A ik algorithms mentioned have the following curiously 
lociil property, They can be L-tescribed roughly 4 s computing dhe 
stored information M as a function of a largedHta set: 

M - A r * (dataset) , 

Now one can imagine algorithms which would use a vast amount 
of temporary storage (lhat is, anah more lhan M or much more 
than is needed to si ore the data set) in order to compute ,tf. Our 
A*k algorithms do not. On the contrary, they do not even use 
significantly more memory capacity than is needed to hold their 
final output, .-1/. They arc even content 1c examine- just one mem¬ 
ber of Lhc data set at a time, with no control over which they 
will fcc next,, and without any subterfuge of aiming the data in¬ 
ternally. 

It seems Lo us that this is Hr interesting properly of compulation 
Lhnt deserves Lo he studied in its ow-n right. It is striking how 
many apparently '’global' 1 properties of a datn set can be com¬ 
puted "iriere-mentillly” in this sense. Rather than give formal 
definitions of thc<e terms,. we shall illustrate them by simple 
examples. 
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SSSSESSSS^! 

?-“-£Srir5s«s 

Sb«SSSt 2 m» 

the data is presented only one*. 

Wfc^ISr" - ■" *• “■ " £ “ —- 1 * “ 

the median ■ 

i ■ H. [ vm i, r fir'Ll siefcr however, that an incremental compu- 

1«»“''"Tl ulu^r*™ k“«in !i«* tb.t»» ««« ">»« 

sssshhs 

ucceplnhle probability Pf error, 

.... iu„ K „,,n uiinw ihtsr drastic e*chun£*s. 

. ffiSSS :»£=» s:... 

set Li: 

|,he numbers in the data S*l catenated in rnimerktdly 
htercasing order, form ii prime numberl, 

ssTrcr-'-iH-SS 

dedsions About wbelher concatenation* of rticmbe 
scL a re bull i me tape* i™ Tnrin £ -n adiines. 







